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ABSTRACT 


The military drawdown and budget cutbacks have created a greater emphasis on 
shorter, faster, and cheaper ways to do the mission. Modulation simulators have become 
a critical component in the evaluation and testing, and integration of new network, 
communications, and command and control technologies and applications. 

This thesis evaluates five current commercial off the shelf products: OPNET 
Modeler by MIL 3 Inc.; COMNET II designed by CACI Products Company’s; Extend by 
Imagine That Inc.; Workbench created by Scientific Engineering Software Inc.; and G2 
from the Gensym Corporation. Each of the products is evaluated using twenty-five 
primary evaluation criteria. The evaluations include current costs, completeness, user 
interface and post analysis support methods, and they describe the basic features and 
distinctive characteristics of each product. 

The capabilities of the products are compared, providing the reader with the 
knowledge to make an informed product selection based on the user’s needs. A matrix of 


the products and the evaluation criteria provides a quick overview of the analysis. 
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EXECUTIVE SUMMARY 


The drawdown and budget cutbacks have created a situation where the military 
must use alternate means to evaluate existing and emerging communications, computer 
and information systems. It is no longer feasible to build computer, communications and 
information systems solely for testing and evaluation purposes. Money, equipment, 
soldiers and time are not available to disconnect existing information systems for testing 
and evaluation. 

It is standard practice to utilize software simulation, including hardware and man- 
in-the-loop, to expedite development, testing, and evaluation of computer, 
communication, and information systems. This thesis examines the current state-of-the- 
art modulation simulators, so-called because signal modulation is required to 
communicate information. 

Modulation simulation is a computerized method for modeling a current or future 
communications, computer, or command and control system. Modulation simulators 
assist the user in designing, evaluating, testing, and reconfiguring a system. It is much 
less expensive to model a new system with a modulation simulator than to physically 
build a demonstration example. Modulation simulators allow for parameter variation, 
protocol variation, load testing, and reconfiguration all within the computer. Modulation 
simulators allow users to examine emerging system’s interfaces with current military, 
commercial and joint networks, and they provide a method for testing futuristic 
technologies. Additionally, they enable the user to evaluate network performance, to test 
the effects of system outages, to stress the system under peak loading, and to see how a 
network would react as equipment is moved, simulating the flow of the battle or OOTW 
(Operations Other Than War). 

An analysis of five of the current commercial off the shelf software modulation 
simulator products provides the modeler with the information needed to make an 
informed product selection. The five products, OPNET Modeler developed by MIL 3 
Inc., COMNET III by CACI Products Company, Extend created by Imagine That Inc., 
Workbench by Scientific and Engineering Software Inc., and G2 from the Gensym 


Corporation, were evaluated against twenty-five various criteria. The following table 


X1 


provides a non-weighted comparison/evaluation of the products and some general 


overview information. A rating of 1, the lowest, to 5, the highest, has been established 


for each evaluation criteria. 


General 
Usage 


Vendor 


Platforms 


Price 
Commercial 
University 

Accessibili 
Completeness 
Configuration 
Management 

Cost 
Data Input 
Methods 
Data 
Reduction 
Documentation 
Ease Of Use 
Experimental 
Design 
Extendibili 
Extensibility 
Of Results 
Fideli 
Flexibili 
Interoperability 
Miscellaneous 

Operational 
Environment 
Post Analysis 

Support 
Reliabili 

Securi 

Software 





OPNET COMNET III EXTEND WORKBENCH 
MODELER 
Comm Network Simulation System Level 















Simulation 
Tool 


Tool 


Networks, 
Computer 


Planning Tool, 
LAN/WAN 
Systems and 
Applications, 
and Distributed 
Systems 


Compan Inc. 


Sun SPARC, PCs (Windows | PCs (Windows | Sun Micro- 
3.1, NT and stations, IBM, 
Windows 95), HP 
Macintosh 





SGI, Windows }| Windows 95), 
UNIX, DEC, 
HP UX, SGI, 


AIX, Solaris 


$600.00 $3000.00 $350.00 $500.00 


NT 


Xi 





G2 


Intelligent Real 


Time Systems 


Gensym Corp. 


Digital, IBM, 
HP, Sun 
Micro-stations, 
PC’s 
(Windows NT 
and Windows 
95 


$40,800.00 
$500.00 
3 
4 
5 


fj N~ ho 


MINH IRL 


U2 


OPNET COMNET HI EXTEND WORKBENCH G2 


MODELER 
Assistance 
User Interface i 
Validation 5 
Verification 5 5 
Non Weighted 106 100 95 
Totals 


Each of the five products analyzed has special characteristics and qualities. The 
products’ functions and capabilities vary, making each product perfect for a different 
scenario. 

CACI’s COMNET II is the best choice for modeling a wide area network or local 
area network. COMNET III provides all the familiar computer network icons in a user 
friendly environment to enable the user to quickly model and analyze a computer 
network. COMNET II, however, does not model communication or information 
systems. Nor does COMNET III model the innards of a computer. 

MIL 3 Inc. developed OPNET Modeler, which can model computers, 
communication and information systems from the network through the process and state 
levels. Numerous libraries are available to assist the modeler. OPNET’s primary 
disadvantage is its fixed run speed. This can cause extensive time delays during a 
simulation run of a detailed model. 

SES’ Workbench, like OPNET, can be used to model communication, computer, 
and information systems. In addition, Workbench can be used to model numerous other 
types of systems, to include queues, electric flow, water pollution, etc. The ability to 
model more than just communication, computer and information systems provide an 
added flexibility to the modeler. 

Extend, developed by Imagine That Inc., is similar in its capabilities to 
Workbench, in that it can also model any type of system. Extend, however, provides a 
more user friendly environment for the modeler. Besides the documentation’s ease of 


use, the menu setup has a more logical flow. Workbench, through the use of sub- 


X1il 


modules, can model to a smaller level of detail than Extend, and Workbench can handle 
larger sized models. 

G2, also similar to Workbench in its system modeling capabilities, is in a class by 
itself. G2 utilizes an object oriented modeling environment with a structured natural 
language. This product is a great selection for a beginner or non-programmer. Object 
oriented modeling enables the modeler to design the level of fidelity and complexity to be 
modeled. G2 includes a wide range of knowledge bases or libraries, though the labeling 
is puzzling. G2/object oriented modeling is not an easy product/modeling method to 
learn. 

In making a selection, it, also, 1s essential to look at the capabilities and 
characteristics of the modeler. Programming skills may influence the product selection. 
All five of the products are icon and menu oriented. COMNET III provides the smallest 
programming interface, and G2 has an easy to learn structured natural language. OPNET, 
Extend, and Workbench each use a modified version of the C language. Workbench 
presents the user with the largest interface with the model’s code, and provides the user 
with the most programming opportunities. 

The user must take into consideration the various characteristics of the system to 
be modeled and the capabilities of the modeler. This combination must be compared to 
the analysis of the products to correctly select the proper software modulation simulator 


for the mission. 
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I. INTRODUCTION 


A. MILITARY AND MODULATION SIMULATORS 

It is standard practice to utilize software simulation, including hardware and man- 
in-the-loop, to expedite development, testing, and evaluation of computer, 
communication, and information systems. This thesis examines the current state-of-the- 
art modulation simulators, so-called because signal modulation is required to 
communicate information. 

Modulation simulation is a computerized method for modeling a current or future 
communications, computer, or command and control system. Modulation simulators 
assist the user in designing, evaluating, testing, and reconfiguring a system. It is much 
less expensive to model a new system with a modulation simulator than to physically 
build a demonstration example. Modulation simulators allow for parameter variation, 
protocol variation, load testing, and reconfiguration all within the computer. Modulation 
simulators allow users to examine emerging system’s interfaces with current military, 
commercial and joint networks, and they provide a method for testing futuristic 
technologies. Additionally, they enable the user to evaluate network performance, to test 
the effects of system outages, to stress the system under peak loading, and to see how a 
network would react as equipment is moved, simulating the flow of the battle o. OOTW 
(Operations Other Than War). 

Most organizations currently using modulation simulators are developmental, 
testing and evaluation, or acquisition oriented. Some examples include: Directorate of 
Combat Development, Fort Gordon, Georgia; Joint Interoperability Testing Command, 
Fort Huachuca, Arizona; and Defense Information Systems Agency, D8, Arlington, 
Virginia. 

Modulation simulators could, also, be utilized by tactical operations staffs. 
Networks designed for tactical exercises could be modeled, tested, and evaluated for 
trouble spots prior to deployment of personnel and equipment. Networks could be 
tweaked for optimal reliability prior to installation. This would save the military precious 


time and money. 


B. SCOPE 

The focus of this thesis is on commercial modulation simulators. Budget cutbacks 
have reduced military software development resulting in most units turning to 
Commercial Off The Shelf (COTS) products. 

This thesis evaluates five of the current commercial modulation simulators: 
OPNET Modeler 3.0 by MIL 3 Inc.; COMNET III by CACI Products Company; Extend 
by Imagine That Inc.; Workbench by Scientific and Engineering Software Inc.; and G2 by 
Gensym Corporation. Each product is evaluated against twenty-five assessment criteria. 
See Appendix. A score of 1 to 5 (5 is the best) will be assessed for each criteria. The 
objective is to provide the reader with the information required to intelligently select the 
most useful modulation simulator for a specific mission. 

The software product chapters, Chapter I through Chapter VI, include an 
overview of each product and a detailed evaluation of the product based on each of the 
criteria and sub-criteria. Chapter VII 1s the analysis/comparison chapter which 
summarizes the results and provides the reader with a basis for making an informed 
selection. Examples are provided in Chapter VII to assist the reader with how to 
effectively select the correct software product. 

Evaluations are based on a hands-on utilization of each product, tutorials, world 
wide web and user demonstrations, all available documentation, and interviews with 


users, sales representatives, and technical support personnel. 


C. PRODUCT SELECTION 

The software products selected were based upon availability to the author, current 
or proposed usage in the military, and experienced users’ recommendations. Many of the 
products can be used to model a wide range of systems. This thesis concentrates on 
simulating communication, computer and information networks. This includes computer 
Local Area Networks (LANs) and Wide Area Networks (WANs), current military 
communication systems (Mobile Subscriber Equipment, Digital Group Multiplexing 


Equipment, SINCGARS, etc.) and message switching networks. Additionally, the thesis 


examines the software product’s interface capabilities for modeling joint and emerging 


technologies. 





IT. OPNET MODELER 


A. INTRODUCTION 

OPNET Modeler™ by MIL 3 Inc. is a computer aided engineering system for the 
design, simulation, and analysis of communication networks, computer systems, 
applications, and distributed systems. OPNET was introduced by MIL3 Inc. in 1987. 
Over 500 organizations, both civilian and Department of Defense, are currently using 
OPNET (MIL 3, 1987). 

OPNET Modeler has a detailed object-oriented environment. It utilizes a 
windows user interface with icons, drop down menus and text editors for inputting the 
network. OPNET provides the user with the flexibility to model top-down, bottom-up or 
middle-out. 

OPNET provides the user with the ability to model from the Wide Area Network 
(WAN) down through the process and state level. The process and state level is used to 
model the actions, changes, functions, and conditions of the basic components of a 
model. Various degrees of modeling provides the user with the added flexibility for 
modeling current or designing future communication and computer networks, systems 
and applications. 

The following information is gathered from the UNIX on-line documentation, the 
MIL 3 Inc. world wide web page, the MIL 3 Inc. sales pamphlets, UNIX on-line tutorials, 


and telephonic interviews with sales personnel. 


B. ASSESSMENT CRITERIA 


1. Accessibility 

OPNET Modeler can be installed on these platforms: Sun SPARC Solaris 2X; 
Sun SPARC Sun OS; Digital Equipment Corporation OSF/1; Hewlett Packard UX; 
Silicon Graphics IRIX; and Intel based implementations of Windows NT 3.5.1. 

OPNET requires 75 megabytes (MB) of disk space for the software and 32 MB of 
Remote Access Memory (RAM). MIL 3 Inc. recommends 120 MB of disk space and 64 


MB of RAM. OPNET requires access to an ANSI C compiler, which is included with the 


Solaris and Sun packages. 


ee Completeness 

OPNET Modeler is a comprehensive product. It models computers, software, 
WANs, Local Area Networks (LANs), data channels, or any other systems, applications 
or processes the user wishes to create. It includes the ability to model current and future 
technologies. 

OPNET includes these protocol libraries: Ethernet, Token Ring, Fiber Distributed 
Data Interface (FDDI), Asynchronous Transfer Mode (ATM), Frame Relay, X.25, Link 
Access Procedure-Balanced (LAPB), Transmission Control Protocol (TCP), Internet 
Protocol (IP), User Datagram Protocol (UDP), Routing Information Protocol (RIP), 
Distributed Queue Dual Bus (DQDB), High Speed Serial Bus (HSSB), In-vehicle 
Communication Bus, Advanced Mobile Phone Service. Connectionless Network 
Protocol (CLNP), Transport Protocol 4 (TP4), and Global System for Mobile 
Communications are all available from third parties. 

OPNET models multiple client/server applications (email, database, file transfer, 
etc.), peer-to-peer protocol delays, server backlogs, and transaction response time and 
throughputs. It, also, models satellite systems and mobile communications nodes. 

Some military system libraries are available for this product. These are available 


through various Department of Defense agencies. 


3: Configuration Management 

The user is responsible for managing input and output files. Separate probes will 
have separate output files. However, a rerun of a model will overwrite existing output 
files. 

The user can graphically display the Network, Node, or Process levels of the 


model. Each layer provides the user with a different degree of the model’s detail. 


4. Cost 
OPNET Modeler is available commercially for $18,000.00. This includes all of 


the current commercial libraries and 90 days of maintenance and technical support. 


Additional technical and upgrade support is available for $3,000.00 per year. The 
university edition is available for $600.00 per year. 

MIL 3 Inc. offers a free one day training seminar. They, also, have a three day 
OPNET Modeler course available in Washington, DC for $1,500.00. Customized 
training can be requested at the user’s location. The pricing of this option is dependent 


on the location, facilities, and instruction requested. 


ae Data Input Methods 

OPNET,, in addition to using predefined Generator Modules (Ideal and Clock), 
uses stochastic traffic sources, based on a user selected probability function, for creating 
message inter-arrival times, and uses deterministic sources to specify exact times for 
message/packet generation. A regenerative source can be used to create “bursty” 
transmissions. Tables are used to prompt the user for data through out the development of 
the model. 

External Model Access includes the capability to extract data from another model. 


(See Interoperability for additional information) 


6. Data Reduction 

OPNET Modeler provides immediate accessibility to probe (data collection 
method) results. This data can be formulated into graphs and reports. The probes are user 
selected and include node statistics, link statistics, global statistics, and simulation 
attributes. 


The following probability distributions are available in OPNET Modeler: 


e Bernoulli e Erlang e Poisson 
e Chi-Squared e Exponential e Uniform 
e Constant e Normal e Uniform-Integer 


These can each be utilized via menu options. 


Ae Documentation 


On-line documentation is included with the UNIX packages. Hard copy 


documentation can be purchased for $500.00 a copy. 


The documentation is very detailed and informative. It includes a comprehensive 
tutorial, a user’s manual, two volumes on modeling, and two volumes of the OPNET 
Modeler simulation kernels. 


No design specifications were available. 


8. Ease Of Use 
This product is fairly complicated to the new user. OPNET Modeler has such 


detailed capabilities it is difficult for an inexperienced user to quickly grasp all of its 
qualities. 

OPNET Modeler enables the user to model events at the process and state level. 
The user continues with the nodal model and then integrates it all together in a network. 
This provides the user with very powerful capabilities. The user can vary the degree of 
depth of the programming parameters and events. The user, also, has access to the 
entrance and exit criteria of the states. The process and state level is the most basic 
breakdown of an element of a system. This enables the user to establish the specific 
conditions required to enter or exit a state of a system. This access is entirely in Proto- 
C™, a computer language very similar to C. 

The manuals are extremely helpful but are oriented at the programmer level. 
Many modeling concepts are just assumed. 

Set up and run time depend on the model. The basic models, only a few nodes, 
contained in the tutorial required between one and three hours of setup. Each of these 
only took a few minutes to run. 

It is difficult to adjust the speed of the simulation clock. OPNET is designed to 
run as fast as it can. The total execution time is dependent on the total number of events, 
the complexity of each node and state, and the type of platform the simulation is being 
executed on. The user can speed up the simulation slightly by leaving out the debugging 
option during the execution. Also, the simulation can be forced to run at real time by 


establishing a comparison of the game clock with the real clock. 


9. Experimental Design 
This product allows the user to execute iterative runs. Various probes can be set 


up for each of the runs. Separate output data files must be established prior to the 


execution. State and process level modeling enables the user to develop models of 


emerging technologies. 


10.  Extendibility 

OPNET can easily be modified. The user must first access the desired level to be 
altered, then right click on the item to be altered. This provides the user with the 
parameter menu. Additionally, OPNET offers the option of an attributes menu. This 
menu will allow the user to input specific numbers immediately prior to the run’s 
execution. 

OPNET Model uses Proto-C™ coding. Immediate access by the user to the entire 
model’s code is not available. The user has open access to the entrance and exit coding 
for the state diagrams. Also, if an error occurs during compiling or execution, a text 


editor with the subroutine containing the error will be accessible to the user. 


11. Extensibility Of Results 

OPNET Model’s ability to model at the state level enables the user to literally 
create any communication or computer system.. Traffic flow, architectures, software, 
protocols and data channels can be modeled. This program is limited by the user’s 
creativity, modeling and programming skills. Communication and computer icons can be 


modified to model alternate architectures. 


12. Fidelity 


This product provides results consistent with the input parameters and models. 


Models can be developed with various degrees of detail. 


13. Flexibility 

Nodes can simulate movement and outages during run execution. Outages can be 
triggered by counters, probability distributions or the game clock. Movements can be 
simulated via timed outages. 


Once the network is modeled it takes little time to make modifications. 


14. Interoperability 
OPNET has an External Model Access (EMA). EMA is a technique of accessing 


a model external to the OPNET program (MIL 3, 1989-1996, pp. EMA-3). This includes 


creating a model, modifying a model, or just extracting data from a model. This 1s all 
supported via C programming language functions. 

OPNET, also, has an External Compiler Interface (ECT). ECI permits the 
interfacing with third party compilers and linker programs. 


15. Miscellaneous 

Map overlays are available. However, they only provide a concept for the 
geographical locations of WANs, satellite equipment and footprints, and mobile cell 
layouts. There is no integration of equipment with terrain density or elevation. 

Outages can be implemented via triggers, probability distributions or timed 
events. These can be programmed to be permanent or merely timed events. 

OPNET can, also, model atmospheric propagation among satellite links, and 


airborne communication systems. 


16. Operational Environment 
The portability of the product is dependent on the computer platform. All 


platforms available are COTs. 


17. Post Analysis Support 

OPNET Modeler provides a wide variety of reports and graphs. The data 
collected is user selected via probes. Probes are established to collect data on various 
points in the network. Multiple probes can be utilized during the same run. Probe types 
are: node statistics, link statistics, global statistics, and simulation attributes. 

The user can view the probe results with multiple or single output vectors. These 
vectors are selected by the user and displayed as traces, or graphs. OPNET Modeler’s 
analysis tool supports a variety of calculations: histograms, probability distribution 
functions, cumulative distribution functions, and confidence intervals. Additionally, the 
user can create mathematical filters to help evaluate the data. 

This program enables the user to export the results with encapsulated Postscript 


files or TIFF files to desktop publishing packages. 
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18. —_—‘ Reliability 
OPNET Modeler is a Proto-C based product with icon and menu user interfaces. 


It executes in its own run time environment. Repeat runs produce consistent results. 


19. Security 

OPNET provides a restricted access security. Three types of users can be 
established. A basic user can run a model, but can not view the model in OPNET Editor. 
An inspector has basic user permissions plus the ability to “inspect” the model with 
certain interface procedures. A full user has complete access to the model and can edit 


the model via the OPNET Editor. 


20. Software 

OPNET Modeler is an object-oriented program introduced by MIL 3 Inc. in 1987. 
The source software is unavailable to the user. The user can program entrance and exit 
criteria for states with Proto-C. Proto-C is very similar to the C language. The user is 
provided access to some programming code in the event of an error. This is only to assist 


the user in troubleshooting the fault. 


21. ‘Training / Tutorial 

The tutorial is comprehensive and informative. It takes the user step by step 
through the menus and model development of various types of networks. Some of the 
menus were incorrectly labeled in the tutorial, but the correct menu was pictured and the 
correct Steps to reach the menu were given. The tutorial provides the user with a 
intermediate knowledge of the product. Someone with little or no modeling background 
would have a difficult experience using the product after the tutorial. It is written for a 


Semi-experienced modeler/programmer. 


22. Troubleshooting Assistance 

Troubleshooting is difficult. All troubleshooting is oriented to the programmer. 
The documentation provides a basic overview and helps the user to locate the cause of the 
fault. Errors are listed by line of code, which is immediately displayed for the user. 


Various options are available within the animation probe to enhance the user’s 
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troubleshooting capabilities. Assistance is available by phone, email, and through the 


world wide web. 


23. User Interface 

The user interface is a mixture of windows, menus, icons, and text editors. A 
window listing the current functions of the mouse buttons is provided at the bottom of the 
screen. A command window informs the user of the current status of the screen. 
Multiple levels can be open at the same time, and a recirculate button permits the user to 


move freely from level to level. The numerous menus can be confusing to novice users. 


24. Validation 
The tool is capable of providing a very detailed solution. The simulation of 
existing systems, where measurement data was available, produced results consistent with 


the real data. 


25. Verification 
The tool seemed to function as described in the documentation. Instructions 


detailed in the tutorial produced the desired results. 


OF CONCLUSIONS 

OPNET Modeler, while not for the beginner, is a comprehensive tool. It 
provides the user with the ability to model at the state and process level. This provides a 
greater flexibility in the modeling of new technologies. 

OPNET is a sophisticated program for modeling and simulating 
communication systems, protocols, and networks. OPNET provides a friendly 


environment for building and analyzing communication and computer networks. 


1 


Il. COMNET III 


A. INTRODUCTION 

COMNET II™ from the CACI Products Company is a network planning tool 
designed with an object-oriented environment to model Local Area Networks (LANs), 
Wide Area Networks (WANs) and Metropolitan Area Networks (MANs). The network 
is created graphically and requires no actual code programming by the user. 

Network technologies and traffic files can be imported from various systems. 
Animation provides the user with the ability to monitor the flow of the network. Building 
blocks, objects a user is familiar with, which include computer nodes, routers and links, 
may be edited to define specific characteristics of a network. 

The basic steps to build a model using COMNET III are define the topology, and 
establish traffic and computer loads. The network is then verified for correctness and run. 
The game clock and the run duration can easily be altered by the user. Reports, selected 
by the user, are automatically generated during the run. 

COMNET IU allows the user to (CACI, 1997): 

e Predict end-to-end delays, throughputs, and utilization of links, buffers and 

processors 

e Reproduce random and bursty traffic patterns 

e See peaks and valleys of traffic not just snapshots and averages 

e Pinpoint sources of delays and bottlenecks. 

The following information is gathered from the CACI Products Company’s world 
wide web page, COMNET LI Tutorial (Sullivan, 1996), and telephone interviews with 


the sales and support offices. 


B. ASSESSMENT CRITERIA 


1. Accessibility 

COMNET III software currently is available for Windows 95, Windows NT, all 
major UNIX computers, Silicon Graphics Inc., AIX, Solaris, Digital Equipment 
Corporation, and Hewlett Packard UX. The software package is 20 megabytes (MB) for 
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the UNIX or personal computer (PC) platforms. It also requires 64 MB of remote access 
memory (RAM) for the UNIX and 32 MB for the PC. 


2 Completeness 

COMNET III uses nodes and links as the primary model building devices. 
Processor nodes model communication processing devices through time delays 
simulating the execution of software applications and the processing of packets. 
Processor nodes simulate file read and writes and application runs through time delays 
and ““Wait For” commands. 

Router nodes model routers, bridges, switches, hubs, and other devices that have a 
common backplane for moving packets between port buffers. 

Switch nodes model switches, routers, hubs, and other communication devices 
that have insignificant packet moving delays. 


The following links can be modeled by COMNET II (CACI, 1997, pp. “A Quick 


Look At COMNET II’): 
e Aloha e CSMA e FDDI 
e Priority FDDI e Polling e DAMA 
e Point-to-Point e Token Ring e CSMA/CA 
e Token Passing e CSMA/CD 


The Point-to-Point links include ISDN and SONET libraries. 

WANs and LANs are easily modeled through use of subnets and WAN cloud 
icons. These provide the user with various views of the network hierarchy. WAN clouds 
can be used to model frame relay, cell relay (ATM), and packet switching (X.25). 

Inter-machine communications can be modeled if the user knows the parameters 
of the interface. COMNET II does not model communication system networks. 


COMNET III currently does not model any military specific equipment. 


3. Configuration Management 

CACTI Products Company began developing simulations over 35 years ago. 
COMNET II is only a few years old but is widely used throughout the civilian 
community. COMNET III is currently on Version 1.3. It is configured as a proprietary 


execution environment. 
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Users are responsible for managing traffic input and data output. Output files will 
be overwritten during a re-run of a modified model. The entire network is displayed for 
the user. Details of individual nodes can be accessed and parts of the network can be 
“hidden” via COMNET WAN Clouds and Transit Nets. COMNET WAN Clouds and 
Transit Nets are icons which represent portions of the entire network to reduce the clutter 


on the display. 


4. Cost 

COMNET III is $48,500.00 for the commercial version, $2,000.00 for a single 
user university edition, and $3,000.00 for a university department edition. Training costs, 
and maintenance and upgrade costs for the first year are included. Afterwards, the yearly 
commercial maintenance and upgrade fee is $5,100.00 and the university fee is $500.00. 

Additional packages, COMNET Baseliner and the COMNET Advance Features 
Package, are $9,000.00 each. 


2: Data Input Methods 

Users can create discrete data and continuous random variable data through tables. 
The discrete data is based on inputs from the user and a step distribution. The continuous 
data is based on a user selected probability distribution function. COMNET I, also, 
features source objects: Traffic, Message, Response, and Session. Traffic Source is used 
to model a single traffic command. Message Source works with a transport command 
and models protocol-control messages. Response Sources generate replies or 
acknowledgments; and Session Sources are for modeling multiple message, bursts, or 
virtual circuit routed messages. 

Traffic and network topologies can be modeled through external sources. 
TRAFLINK, a CACTI utility, reads external traffic files and formats them for COMNET 
II usage. Network topologies can be extracted from Network Management Systems. 
Third party software, Cabletron Spectrum topology files, HP Open View topology files, 
and IBM Net View 6000 topology files, can extract an existing network’s topology to be 
imported into COMNET. (CACI Products Company, 1995, pp. 210) 
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6. Data Reduction 

COMNET III provides immediate graphs and reports. A Report Menu allows the 
user to turn on and off various reports and statistics for specific items (nodes, links, WAN 
clouds, application sources, message and response sources, session sources, transport 
commands, and setup commands). Upon completion of the run the user can immediately 
“browse” the results. 


The following probability distributions are available in COMNET II: 


e Beta e Geometric e Poisson 

e Erlang e Hyper exponential e ‘Triangular 
e Exponential e Log normal e Uniform 

e Gamma e Normal e Weibull 


Each of these is available via a drop down menu in the parameters windows. 


Ue Documentation 

COMNET III’s documentation consisted of one book. It is oriented towards the 
network manager. It does a great job of explaining the functions and capabilities of 
COMNET III. However, it does not describe how to use COMNET III. It is left up to the 
user to learn how to use the product. 

A tutorial (Sullivan, 1996) was developed by a third party. It provides step-by- 
step instructions on how to use the major features of COMNET II. 

The lack of documentation did not hamper the author’s ability to quickly learn and 
use the basic features of the product. The menus and icons provide the user with an 


extremely user friendly flow for developing a model. 


8. Ease Of Use 

The product is very easy to use. Network topologies are quickly developed using 
drag and drop icons and a menu driven environment. Parameters are altered by double 
clicking on the various network objects. Menu options enable the user to alter the game 
clock speed, animate the run, trace the run events, and run iterative trials. Results are easy 
to view via the “Browse” option in the reports menu. The network set up time is 


primarily restricted by the network knowledge of the user. 
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The personnel at CACI Products Company were friendly and eager to assist. The 
CACI world wide web page (CACI, 1997, pp. “COMNET Support”) provides helpful 
email addresses and phone numbers. COMNET II and generic modeling questions can 
be answered via a customer hotline provided to valid users. Additional modeling 
assistance is available at a cost from the CACI Modeling Group. 

The product is very easy to learn. The thesis tutorial developed by Jeffrey 
Sullivan (Sullivan, 1996) provides a quick overview of all of the basic COMNET II 
features. 

The run times of this product vary with the detail of the simulation. The simple 
networks designs, created using the tutorial, executed in only a few minutes. A detailed 
10,000 node network with multiple events and detailed parameters would take much 


longer. 


9. Experimental Design 

The modeling tool provides the user with the ability to execute trials iteratively 
during a single run. Additionally, the automatic parameter feature enables the user to 
vary the parameters during the iterative executions. This provides the user with a faster 
method for evaluating proposed network changes. The user can modify existing 
parameters to examine alternate computer networks, but COMNET can not model 


communication systems. 


10. Extendibility 

The program itself cannot be modified or extended. The source code is not 
provided to the user. COMNET II does provide the user with a high level, simple 
program interface. The programming is similar to Basic. It provides greater flexibility to 
the user. Counters and triggers can be established to initiate follow on events. An 
example would be when the utilization of a link is greater than 60% switch to an alternate 
link. 

11. Extensibility Of Results 

COMNET I] provides the user with two primary methods for evaluating an 
emerging architecture. First, with the basic, high level programming option which can be 


used to set counters and triggers to evaluate the results of an event; second through the 


17 


use of transit nets. Transit nets provide a more detailed option for modeling transport 
protocols. It enables the user to create a protocol like frame relay via an ATM link. 


COMNET III does not specifically model data channels or software applications. 


12. Fidelity 
The tool reliably provides results consistent with the input parameters. COMNET 
Baseliner, an add on feature, enables the user to model existing networks with realistic 


external data. The detail of the model can only be slightly varied. 


13. Flexibility 
Nodes can be set up to start, stop, or model a delay during the execution. 


Parameters can be easily altered by double clicking on the items icon. 


14. Interoperability 

COMNET Baseliner, an add on feature, enables the user to model existing 
topologies and use realistic traffic files. Actually topology data can be taken via a 
Network Management System, like HP Open View, Network General, Cabletron 
Spectrum, IBM NetView for AIX, and Castlerock SNMP, and imported in COMNET. 
Network traffic data can be gathered from Network General Distributed Sniffer System, 
Frontier Software NETscout, HP NetMetrix, CompuWare EcoNET and 3COM 
LANsentry. This data can then be imported into COMNET I via COMNET Baseliner. 
This importing capability provides the user with the ability to model a current network 
and real data to run on it. This model can then be tested to find any network problems, 
evaluated for interfaces with new equipment, tweaked to find the peak performance 
design, etc. 

A small amount of traffic gathered can be emulated to create a large database of 


“realistic” data. This prevents the user from having to gather hours of real data. 


15. Miscellaneous 

Outages can be caused or enabled through parameter settings and/or by 
establishing triggers and counters. Outages can be for a predefined time, simulating a 
nodes movement; for a random time, simulating a maintenance outage; or permanently, to 


simulate equipment destruction. 
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Frequency and terrain options are not available with COMNET II. 


16. 


Operational Environment 


This product is available for Windows 95, Windows NT, UNIX stations, SGI, 
AIX, Solaris, Dec, and HP UX systems. All these systems are COTs and easily 


obtainable to users. Portability depends on the platform the software is installed on. 


a. 


Post Analysis Support 


COMNET III provides a wide variety of reports. Each of these is available for 


viewing immediately after the run execution. Reports must be turned on by the user for 
the data to be collected. The following reports are available within COMNET III (CACTI, 
1997, “COMNET II Features List’): 
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Node Reports: Processor and disk utilization; Received message counts; 
Session level; Call counts; Call level; Buffers by node; Buffers by port 

Link Reports: Channel utilization; Utilization by application; Utilization by 
protocol; Frame Size; Collision Statistics; Token ring statistics; Session level; 
Call counts; Call level 

WAN Cloud Reports: Message and Response Source Reports; Message delay; 
Message delivered delay; Packet delay 

Cell Source Reports: Blocked call counts; Disconnected call counts; 
Preempted call counts 

Transport Command Reports: Message delay; Packet delay 

Session Source Reports: Message delay; Message delivered delay; Packet 
delay; Setup delay; Session Length; Setup counts; Transport Layer 

Snapshot Reports Node reports; Link reports; Cloud virtual circuits 

Statistics file for export to other analysis: Real-time plots; Post-processed 


plots; Percentiles; Raw data plot file 


Reliability 


The product is coded primarily in a high level, icon drag and drop language. It 


executes in its own run time environment. Modeling errors may cause the experiment to 


suddenly terminate. Diagnostic information is available. 
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19. Security 


No security options are available. 


20. Software 

CACTI Products Company has been a simulation leader for 35 years. COMNET Ii 
Version 1.3 was fielded in December 1996. The source software code is unavailable to 
the user. The product is a drag and drop icon and menu based environment. It has a 
simple flow for network development. CACTI distributes minor releases throughout the 
year and major releases periodically. Over 2000 various universities and companies are 
currently using COMNET III (CACTI, 1997, pp. “A Quick Look at COMNET H?’).. 
COMNET III is an object-oriented software application. 


21. Training / Tutorial 

CACTI Products Company provides a free three day overview/orientation training 
to new users. Additionally, the tutorial (Sullivan, 1996) is a great beginners manual. 
Training is primarily at the beginner level. COMNET II is not a difficult program to 


learn. 


pipe Troubleshooting Assistance 

Animation and event tracing are available to assist the user in network flow 
monitoring and troubleshooting. All valid users, those having paid the yearly upgrade 
fee, are provided with a Customer Support Hotline. CACTI, for a cost, has a Modeling 
Group to assist companies in developing a model for a network. Assistance is available 


via email and the world wide web. 


23. User Interface 

The product is very user friendly. The network topology can be created via drag 
and drop icons or with menu options. Once the topology is established, parameters may 
be altered by double clicking on the items. A parameter window will appear with drop 
down menus to assist the user. The verify option provides a detailed list of all errors 


present in the network topology or basic flow. 


20 


The user may alter the run time parameters (clock speed, run iterations, animation, 
event trace) through a simple menu. Post analysis support allows the user to easily 


“browse” the reports and statistics. 


24. Validation 

The simulating of existing networks where measurement data is available 
produced results consistent with the real data. 

COMNET Ii provides the user with a basic to intermediate analysis of the 
network and its interfaces. This is dependent on the parameters and events established by 


the uSer. 


25. Verification 

Upon completion of the model, the product does a quick verification of the 
network connections. 

The tool seemed to function as described in the documentation. Instructions 


detailed in the tutorial produced the desired results. 


c. CONCLUSIONS 

COMNET II is a simple, straightforward, comprehensive network planning tool. 
User training is easy, and icon-based network designing allows for quick construction and 
reconfiguration. COMNET III provides a method for analyzing and optimizing 


computer-based networks. 
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IV. EXTEND 


A. INTRODUCTION 

Extend, developed by Imagine That Inc., is an advanced simulation tool designed 
to develop dynamic models of real-life processes. It enables the user to: predict the 
course and results of certain actions; understand why observed events occur; identify 
problem areas before implementation; explore the effects of modifications; confirm that 
all variables are known; evaluate ideas and identify ineffectiveness; gain insight and 
stimulate creative thinking; and communicate the integrity and feasibility of plans 


(Imagine That Inc., 1995, pp. 4). 


B. ASSESSMENT CRITERIA 


1. Accessibility 

Extend is available for Windows 3.1 or later, Windows 95, Windows NT, 
Macintosh 6.7 or later, and Power Macintosh System 7. 10 MB of hard disk space are 
required for the Windows version and 8 MB of hard disk space are required for the 
Macintosh version. Both platforms require 4 MB of RAM but 8 MB are recommended. 

Models built on either platform can be run on the other platform. For example, if 
you build a model or library on a Macintosh computer, it can easily be read on a 


Windows computer. 


Zi Completeness 

Extend uses icons or building blocks to build models. 15 libraries of various 
building blocks are included with Extend. The building blocks represent the smallest 
level or function of a model. The user must build computers, communications 
equipment, systems, etc. from the basic functions and flows. This requires the user to be 
proficient in modeling. 

Users can create new icons which can be saved in the libraries. They can also 
save models as templates to be used in other models. This provides the user with the 
power to create the computers and equipment to be reused in later models. Additionally, 


a list of libraries available through third parties is available through Imagine That, Inc. 
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3: Configuration Management 

Imagine That Inc. was founded in 1987 to market Extend software. Extend is 
current on version 3.2. 

The user is responsible for managing input and output. Extend permits 
hierarchical modeling. The user via levels (subroutines) can make a complex system easy 


to build and easy to understand. 


4. Cost 

Extend is available for $695.00. The university price is $350.00. Technical 
support is free provided the user has purchased the most recent upgrade version. Today’s 
current upgrade from version 3.1 to version 3.2 is $99.00. 


Imagine That Inc. does not offer any in house training programs. 


SF Data Input Methods 

Extend includes an input random number menu, and an input data menu. The first 
provides the user with a distribution selection, and the latter permits the user to enter 
specific data values. 

Extend uses text files for importing and exporting data. Text files can be created 
in another application, like a spreadsheet, word processor or database, then read into 
Extend, and vice versa. This is easier for inputting lists of files and for manipulating 


simulation results. 


6. Data Reduction 

Plotters are built into the model to gather statistics. Results of the plotters can be 
immediately displayed on the screen. 

The following are available to gather statistical information from the results: D- 
Binomial, D-Exponential, D-Gamma, D-Log Normal, D-Pascal, D-Poisson, Gaussian, 
Mean, Random, Random Get Seed, Random Real, Random Set Seed, Standard Deviation 
Population, Standard Deviation Sample, Statistical Value. (Imagine That, Inc., 1995, pp. 
287) 
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7. Documentation 

Extend has one book for its documentation (Imagine That, Inc., 1995). It is very 
comprehensive. Chapters 1 through 3 consist of a tutorial of Extend’s basic features. 
Chapter 4 is an overview of continuous and discrete modeling. Chapter 5 discusses all of 
Extend’s libraries. Chapter 6 describes the methods to get data into and out of Extend. 
The rest of the book provides a more in depth look at Extend, modeling, how to build 


icons, and a list of common errors and their causes. 


8. Ease Of Use 

Extend is fairly easy to use. Models are constructed with building blocks via 
windows and menus. Double clicking on a building block provides access to the items 
attributes. 

Extend’s biggest challenge is the user must know the intricate details of the 
system or equipment to correctly model the functions and flow. 

Set up time is dependent on the depth of the model. The basic examples provided 
in the tutorial took about 15 minutes to create and only a few minutes to run. 

Extend allows the user to vary the run time, and the step speed or clock speed. 


The user can, also, vary the model while it is running online. 


9: Experimental Design 
Extend supports Monte Carlo and batch mode simulations. Iterative runs can be 
conducted. The plotters will only maintain the most recent four plots. Extend’s basic 


features enable the user to model emerging systems. 


10.  Extendibility 

Extend primarily uses icons or building blocks to construct models. Extend’s 
source code is the ModL language. It is based on the language C. 

Extend permits the user to create their own building blocks or modify existing 


ones. The user can use “scripting” to design code for the model. 
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11. Extensibility Of Results 
Extend allows the user to create building blocks. These can be used to create 
virtually any communication or computer system.. Extend’s extensibility is limited to the 


modeling and programming capabilities of the user. 


12. Fidelity 


The tool reliably provides results consistent with the input parameters and the 
model. The user’s ability to create building blocks could effect the behavior of the 


model. Models can be developed with various degrees of detail. 


13. —‘ Flexibility 
Extend provides the ability to alter the model both on and off line. The user can 
simulate the time lost for the movement of a node or for maintenance down time through 


the use of time delays, blocks, and triggers. 


14. Interoperability 

Extend has cross-platform capabilities between Windows and Macintosh. Extend 
allows users to copy data from text entry fields and data tables into other applications. 

Extend permits the user to create building blocks to communicate with scientific 
equipment and other hardware devices. Extend, also, provides two methods for accessing 


code written in another language: XCMD (for Macintosh) and DLL (for Windows). 


15. Miscellaneous 

Outages can be caused or enabled through delays, triggers and blocks. Outages 
can be for a predefined time, simulating a nodes movement; for a random time, 
simulating a maintenance outage; or permanently, to simulate equipment destruction. 


Frequency and terrain options are not available with Extend. 


16. Operational Environment 
This product is available for Windows and Macintosh systems. All of these 
systems are COTs and easily obtainable to users. Desktop and notebook computers are 


both very portable. 
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17. Post Analysis Support 

Extend has a wide variety of plotters available to display output: Discrete Event 
(DE) Error Bars; DE Multi-Simulation; DE; Error Bars; Fast Fourier Transform; 
Input/Output; Multi-Simulation; Histogram; Scatter; Scatter (4); Strip; and the Worm. 

The user can, also, select blocks to record data from for a report. These reports 


only display final simulation values. 


18. _—Reliability 
Repeated runs of models based on constant inputs produce the same results. 


Extend executes in its own run time environment. 


19. Security 


No security features are available. 


20. Software 
Imagine That Inc. developed Extend in 1987. Extend is written in ModL, a C 


based language. The user creates models by connecting building blocks to simulate the 


basic functions and flows of the system. 


21. Training / Tutorial 

Imagine That Inc. does not offer any in house training courses. There are some 
third party training courses available for a cost. 

The first three chapters of the Extend users manual are a comprehensive tutorial. 


They provide the user with a complete overview of Extend’s major features. 


22. Troubleshooting Assistance 

Animation and event tracing are available to assist the user in network flow 
monitoring and troubleshooting. Common error messages and their causes are listed in 
an appendix. 

Extend provides phone, email, fax, and world wide web technical assistance. Free 


technical assistance is available to all users modeling with the latest Extend upgrade. 
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23. User Interface 
The product is very user friendly. Extend’s user interface is through building 
blocks and menus. The user can quickly and easily build a model by selecting building 


blocks from the menu of libraries, and then connecting the blocks together. 


24. Validation 
Extend’s degree of accuracy 1s based on the users design. A properly designed 
model produces realistic results. 


The user creates the model to the depth of answer required. 


25. Verification 
Extend will verify the basics of the code during compilation. 
The tool seemed to function as described in the documentation. Instructions 


detailed in the tutorial produced the desired results. 


GC. CONCLUSIONS 

The program Extend is not difficult to learn. The user must be able to model the 
basic functions and flows of each element and combine them to build the model. 

Due to Extend’s purpose of modeling real life processing in a wide variety of 
areas, many of its features are generic and not communication and computer specific. 
This does not restrict the program’s ability to model communication and computer 


networks but instead expands the user’s modeling options. 
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V. WORKBENCH 


A. INTRODUCTION 

Workbench developed by Scientific and Engineering Software, Inc. (SES) is a 
general purpose modulation and simulation tool. Workbench is a collection of software 
tools used to design and evaluate sophisticated systems of various types. It’s major 
components are: SES/design ™, a graphical editor; SES/sim, a translation and simulation 
module; and SES/scope, an animated simulator. 

Workbench is an object-oriented product. Models are built with a top down 
construction, beginning with the catalog level, then the module and sub-module levels. 

Workbench is designed to model the essential characteristics of a system, while 


ignoring the unimportant details. It focuses on the high-ievel system architectural issues. 


B. ASSESSMENT CRITERIA 


1. Accessibility 
Workbench is available for the Sun, IBM, and Hewlett Packard UNIX platforms. 


De Completeness 

Workbench consists of a variety of module and sub-module building blocks/icons. 
These building blocks can be used to create models of current and future computers, 
software, and communications equipment. The models are constructed by connecting the 
build blocks or nodes with arcs. Workbench permits the user to create icons. These icons 
must be based on a basic building block but can be stored in a library for future usage. 

The basic building blocks or the icon palettes for Workbench are: Resource 
Management Nodes: service, delay, resource, allocate, release, create, destroy, block and 
set; Transaction Flow Nodes: source, sink, fork, join, split, loop, interrupt, resume, and 
branch; Sub-Module Management Nodes: enter, call, sub-model, and return; and 
Miscellaneous Nodes: user, declaration, and super. The catalog level provides the user 
with these building blocks: main module, generic module, remote module, interface 
module, object file, cosim interface, archive file, user make rules, remote catalog anda 


palette icon. 
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Once a module is bui: it can be archived for use in future models. This enables 
the user to create a library of frequently used items. 

Workbench includes a variety of queuing rules. These include priority rules 
which allow no priority, preemptive, and non-preemptive priorities. The time rules 
include: first come, first served; last come, last served preemptive resume; processor 


sharing, and round robin. 


Oe Configuration Management 

SES, founded in 1971, has evolved from a consulting and research organization to 
a global software and consulting firm. For more than 25 years SES has been providing 
simulation technology to its customers. 

The user is responsible for managing input and output files. User’s can specify 
which nodes to gather statistics. A rerun will overwrite previous output files. 

A user can view the modules and sub-modules developed during the building of 
the model. The main module is typically designed to be a complete diagram of the 


network or system being modeled. 


4. Cost 

A SES sales representative will help design a software package of Workbench and 
its associated files to meet the users needs. Workbench’s starting commercial price is 
$23,500.00. The university price is $500.00. The government price is 20% of the cost of 
the civilian priced package. Technical support and upgrade costs are 15% of the original 
package price. 

Training is available at SES Regional Sites for $8,000.00 per person. This is for 
the three day introductory course. 

The Network Framework Library, consisting of: X.25, TCP, ATM, FDDI, SNA, 
Client/Server and Ethernet, can be purchased for $5,000.00 (civilian price). SES, also, 
has a huge list of partially developed models. These models are free and can be requested 


as needed. 
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ny Data Input Methods 
Users can input data internally via tables. External interfaces include: C and C++ 
programs and Objectbench files. The users must establish the link or interface between 


Workbench and the external functions. 


6. Data Reduction 

Workbench provides immediate access to statistics gathered during a simulation 
run. The user can select global or specific statistics to be gathered. The standard statistics 
that Workbench gathers are: inter-arrival, lifetime, population, q-population, q-response, 
quantity, response, and utilization. Users can, also, create user-defined “collect” 
statements in C language to accumulate additional continuous or discrete statistics not 
ordinarily collected by Workbench. 

Workbench includes these probability distributions: empirical, erlang, 
exponential, gamma, hyper-exponential, normal, triangular, uniform, binomial, 


geometric, I-empirial, I-uniform, and poisson. 


ie Documentation 

Workbench’s documentation is quite extensive. The Quick Start Manual provides 
a basic overview of the most commonly used features and includes an example or mini 
tutorial on creating, compiling and animating a simple model. Creating Models, 
Simulating Models, SES/sim Language Reference, Technical Reference, Query 
Interpreter and Language Reference, Cosim Reference, and an Installation Manual are, 
also, included in the Workbench documentation. 


No design specifications were available. 


8. Ease Of Use 

The basic features of Workbench are fairly easy to learn and use. Menus and 
icons create a user friendly environment that provides the user with a graphical interface 
to modeling. Workbench, however, is not designed to model any specific type of 
network. It is a general modeling tool. What this means is the libraries and icons have 
basic low level functions. The user must create anything specific needed for the model. 


The user needs to be a proficient modeler to get the maximum usage of this software. 
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Workbench uses atop down modeling approach. The user establishes a catalog 
with a main module and possibly some other generic modules. Each module is then 
developed further as a sub-module. The user can “travel” back and forth between the 
levels of the modules and sub-modules. This level approach allows the user to break the 
model down into small manageable pieces. 

The user manuals are easy to follow. There is an index in the back of each book 
of the documentation. The index provides a quick reference to all of the documentation 
manuals. The manuals provide details and execution steps for most of the topics. 

Once again run time and set up time are dependent on the detail of the model. 
The user must design the model to the accuracy desired in the solution. Workbench uses 
multithread processing to simulate numerous events occurring simultaneously. The run 
length and run interval time or clock speed, can be altered prior to the execution of the 


simulation. 


9. Experimental Design 
Workbench does not offer an iterative run capability. Workbench’s basic features, 
modeling with sub-modules, enables the user to model emerging system by breaking 


them down into their smallest elements. 


10. Extendibility 
Workbench uses a C based language. Users can add C language procedures to be 
included in the simulation program. These additions enable Workbench to be extended 


as necessary. The user can access the code developed for the model. 


11. Extensibility Of Results 

The ability of the user to add C programming statements, also, enables the user to 
create new and alternative computers, communication equipment and any other type of 
system. This capability is limited by the user’s ability to program in C language. 


SES offers, for a price, extension products. 


12. ‘Fidelity 
This product seems to provide realistic results consistent with the input 


parameters and model. However, Workbench is graph-oriented programming, which is 
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dependent on user definitions for its behavior. Models can be developed with various 


levels of detail. 


13. Flexibility 
Nodes can be delayed or blocked, simulating movement or destruction, through a 
time activation, probability distribution, or a trigger. Each of these must be created as 


part of the model. 


14. Interoperability 
Workbench can interoperate with most programs created in C. C programs can be 


imported into the model and run as part of the simulation. 


15. Miscellaneous 

Workbench does not offer any terrain or frequency features. 

Outages must be developed as part of the model. Delays and blocks can simulate 
temporary and permanent outages. Time delays, probability distributions and triggers can 


be established to create and fix outages. 


16. Operational Environment 
Workbench is only available for UNIX platforms, and thus are not very portable. 


The platforms are commercially available. 


17. Post Analysis Support 

Workbench provides textual reports and machine readable output. The reports 
can be displayed on the screen or printed. The reports consist of: a cover page, two 
optional user-generated output sections, a report header, a detailed statistical report, an 
optional node summary statistical report, and an ending banner. The machine readable 
output can be post processed using most commercial spreadsheets or word processors. 
The user can, also, print the graphical representation of the model. 

Available only on SUN platforms, SES/graph is a small easy to use graph 
program. It is a front end for a program called grafit ® created by Graphicus, Inc., 
Kirkland, Washington. (SES, 1994-1995, pp. A-1). SES/graph enables the user to 
display and plot statistical data produced during the model’s run. These graphs can be 


displayed in bar charts, pie charts, line charts, spline charts, or a step function chart. 
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18. Reliability 


The product executes in its own run time environment. Debugging features are 


included. 


19. Security 


Workbench does not offer any security features. 


20. Software 
Workbench is based on the C language. Workbench uses an object-oriented 


environment. The user can access and alter the code developed for the model. 


21. Training / Tutorial 

Workbench training is available at SES Regional Sites. Customized training can 
be requested, for a cost, to be conducted at the user’s site. Introductory and advance 
classes are available. 

Workbench’s documentation includes a basic tutorial. It is a quick overview of 


the program’s primary features. 


22. Troubleshooting Assistance 

Workbench has animation and trace capabilities. The user can view the code by 
accessing the model’s file. 

SES offers assistance via a 1-800 telephone number, email, the world wide web, 
and through regional sites. Frequently asked questions are available through the world 


wide web to registered users. 


23. User Interface 

The user interface is graphic oriented. The user through menus, icons, and arcs or 
connection tools, creates a graphic picture of the model. The flow of the menus can be 
confusing to the novice user. The user can modify the parameters of the nodes and can 


add C programming to augment the simulation. 


24. Validation 
Workbench provides the user with the option of modeling to the small details or 
on a larger scale. Models programmed via code will depend on the users modeling ability 


for validity. 
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25. Verification 
No verification features were identified. The product seems to function as 


described in the manual. 


C: CONCLUSIONS 

The basic commands for Workbench are easy to learn. Workbench requires the 
user to be a proficient modeler. The user must be able to understand the basic functions 
of the system they are modeling. Workbench, also, places heavy emphasis on 
understanding programming. Workbench is a good tool, however, it is not designed for 


the beginner. 
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A. INTRODUCTION 

G2, from the Gensym Corporation, is designed for developing intelligent, 
knowledge-based, real-time applications. G2 is typically utilized for complex situations 
that require: monitoring, diagnosis, and alarm handling; scheduling and logistics; 
supervisory and advanced control; process design, simulation, and re-engineering; 
intelligent network management; and decision support for enterprise-wide operations 
(Gensym, 1995, pp. 4). 

Each intelligent application developed or modeled is known as a knowledge base 
(KB). KBs are developed in a graphical and menu oriented environment using Object 
Oriented Modeling. Gensym Corporation uses Object Oriented Modeling as the bases of 
the G2 product. 


B. ASSESSMENT CRITERIA 


1. Accessibility 

G2 can be installed on UNIX, VMS, Windows 95, and Windows NT platforms. 
These include Digital Equipment Corporation, Hewlett Packard, Sun SPARC stations, 
Silicon Graphics, and IBM Reduced Instruction Set Computers. G2 requires 30 MB of 
hard disk space and 48 MB of RAM. 


2: Completeness 

G2 enables the user to model computers, software, and communication equipment 
through the use of Object Oriented Modeling. Specific KBs must be developed by the 
user. 

G2 provides the modeler with the capabilities to create not only the model but the 
interface that the operator will have with the simulation. The operator can be prompted 
for input via toggle switches, text inputs, or interactive buttons. The modeler can create 
an interactive on-line simulation environment. 


The Army Artificial Intelligence Center is currently using G2. 
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3: Configuration Management 

Gensym Corporation was founded in 1986. Gensym Corporation currently 
provides services to over 30 countries which include petroleum companies, chemical 
companies and other industrial companies. 

The user is responsible for managing input and output files. G2 graphically shows 
the objects within the designed system. Data encapsulation or hiding is available to 
reduce clutter and access to some information. Object relationships can be depicted by 


showing the workspace module hierarchy. 


4. Cost 

The current commercial cost for the G2 server license with Telewindows is 
$36,000.00. A minimum of one user license must be purchased at $4,800.00 (User license 
prices decrease after the eleventh user). The university cost is $500.00. The GSA cost is 
31% of the commercial cost. 

Customer service support is available for a yearly fee of 15% of the commercial 
list price ($6,120.00 for a single user package). 

Free one and one half day workshops are available at select regional sites. G2 
Part I and Part II training, each a four and one half day course, is available for $2,295.00 


each. 


a Data Input Methods 

G2 includes internal and external data source, or data server, features. G2’s 
internal data servers are the G2 inference engine and the G2 simulator. The G2 inference 
engine “concludes” or calculates an answer from specific formula objects. The G2 
simulator receives a value from a simulation formula. 

External data server methods include data received from other G2 processes, from 
GFI bridge applications, and from GFI files. (See Interoperability sub-section for more 


information) 


6. Data Reduction 


G2 does not offer any standard statistical features. 
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Fe Documentation 

G2 provides a complex assortment of documentation. The titles of the manuals 
are very misleading as to the content to a beginning user. These manuals include: G2 
Reference Manual, Upgrading Your Knowledge Bases, G2 Systems Procedures Reference 
Manual, Telewindows User’s Guide ,and G2 GUIDE User’s Guide. G2 documentation 
is very complicated. It is oriented toward a proficient object oriented modeler. 

G2 does not offer any tutorial. 


Design and modeling documentation is not available. 


8. Ease Of Use 

G2 is not easy to learn. It is entirely object oriented. Existing KBs are not labeled 
well enough to know what they contain. The user can create KBs of computers and 
communication equipment and save them for reuse. 

The manuals provide guidance on the G2 functions and features but do not cover 
any Object Oriented Modeling. 

Set up time is dependent on the detail of the model, the related KBs available to 
build upon, and the users ability to use object oriented analysis and design. 

G2 has three options for running a simulation: real time, simulated time, and as 


fast as possible. The model and the inputs can be altered while the simulation is running. 


9. Experimental Design 
G2 does not offer any iterative run capabilities. Object oriented modeling enables 


the user to create objects and attributes to model emerging systems. 


10. Extendibility 

G2 models can be altered on line or off line. G2 uses a structured natural 
language, similar to COBOL. The language uses ordinary human statements, so the user 
does not need to be a proficient programmer. G2’s natural language, also, supports 
Japanese, Korean, and Russian. 

A text editor is available for the user to input specific statements, rules, functions, 


attributes, etc. 
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11. Extensibility Of Results 
G2 and Object Orient Modeling provide the user with a flexible environment 
capable of modeling virtually any computer or communication system the user can 


imagine. The product is limited by the users ability to analyze and design a model.. 


12. Fidelity 
The tool reliably provides results consistent with the input parameters. Models 


can be developed with various degrees of detail. 


13. Flexibility 
Nodes can be moved both on and off-line. The user can establish time delays, 


triggers or interactive inputs to initiate the move or outage. 


14. Interoperability 

G2 provides networking and interfacing through Telewindows, G2 to G2 
Interface, G2 Standard Interface (GSI), and G2 File Interface(GFI). Telewindows allows 
more than one user to access, or remote view, a running G2 process. G2 to G2 Interface 
lets two or more G2 process to connect to share data. GSIis a toolkit for developing 
bridges between G2 and external systems. GFI allows G2 to read and write external data 


files. 


15. Miscellaneous 
Outages can be simulated with time delays, triggers, or interactive input by the 
operator. 


Frequency and terrain options are not available with G2. 


16. Operational Environment 
This product is available for UNIX, VMS, and Windows platforms. All these 
systems are COTs and easily obtainable to users. Portability depends on the platform the 


software is installed on. 


17. Post Analysis Support 
G2 features include readout tables, dials, meters, freeform tables, charts, graphs 
and trend charts. Readout tables, dials and meters depict changing values during a 


simulation run. Freeform tables are similar to spreadsheets. Charts plot numerical data 
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series. Charts include data points, bar charts and column charts. Graphs are being 


replaced by trend charts. Trend charts plot times series and historical data. 


18. Reliability 
The product executes in its own run time environment. Results are consistent 


with the inputted model. 


19. Security 

G2 provides network security features. These features include: connect, read, 
write, execute, and inform. Connect prohibits or enables other users from accessing a 
KB. Read and write prohibit or enable other users from reading or writing variable 
values. The execute command deals with remote procedure calls, and the inform 


command prohibits or enables other users from sending messages to the operator. 


20. Software 

G2’s structured natural language provides the modeler with a user friendly 
program interface. The modeler can use ordinary human language statements to input 
specific commands for the model to execute. G2’s language and modeling environment 


are object oriented. 


21. Training / Tutorial 

G2 offers numerous training courses: Introduction, G2 Part I, G2 Part Il, and many 
classes for supporting products. Courses provide a wide variety of information on how to 
use G2. The combine syllabus for the Introduction, and G2 Parts I and II indicates that 
upon completion of the training the user will have an intermediate knowledge of the 
product. 


No tutorial is available. Some demonstration KBs are included. 


22. Troubleshooting Assistance 

G2 has been rated number one in customer support by Control Magazine for the 
last two years (Gensym, 1997, pp. “Services”). Engineers are available, by telephone, 
Monday through Friday, to assist with installation, product use, problem diagnosis, and 


configuration questions. 
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G2 Help Link is a 24 hours a day, seven days a week, automated help service 
available through the G2 world wide web home page. The Gensym Users Society (GUS) 
provides technical communications between Gensym customers and the Gensym 
Corporation. Help services are, also, available through email. 


Animation is not available. 


23. User Interface 

The G2 user interface is menu oriented. The number of menus can be 
overwhelming to the beginning user. Menus lead to construction of objects, classes, 
connections, etc. Once an item is created, it is defined through a table. Most table entries 
are prompted with a list of available options. Some entries allow the user to provide 
input into the model. For example: variable and object names, or the use of structured 


natural language to input exact specifications for an object. 


24. Validation 
Users can model to the depth desired. The user must use correct object oriented 


analysis to properly define the system to be modeled. 


25. Verification 
The product functioned as described in the manuals. Gensym Corporation has 
sold more than 7500 product licenses to over 500 organizations worldwide (Gensym, 


1997, pp. “Company’”). 


c; CONCLUSIONS 

G2 is based on object oriented analysis and design. “Object oriented analysis is 
based upon concepts that we first learned in kindergarten: objects and attributes, classes 
and members, wholes and parts.” (Pressman, 1992, pp. 239) 

The product, while initially intimidating, is a very powerful tool. The structured 
natural language enables a non-programmer to have the features and capabilities available 


to create a detailed model of a complex system. 
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Vil. ANALYSIS 


A. INTRODUCTION 


This section compares the five software products. A rating of 1 to 5 has been 
established for each evaluation criteria based on the sub-criteria contained in the 
Appendix: 

1 - no sub-criteria satisfied 

2 - less than half of the sub-criteria satisfied 

3 - half of the sub-criteria satisfied 

4 - more than half of the sub-criteria satisfied 

5 - all of the sub-criteria satisfied. 

Additionally, general usage, vendor, available platforms and the average price have been 


included. 


B. COMPARISON 


Table 7-1 shows a non-weighted comparison of the software products. 


OPNET COMNET II EXTEND WORKBENCH G2 
MODELER 
General Comm Network Simulation System Level Intelligent Real 


Usage Networks, Planning Tool, | Tool Simulation Time Systems 
Computer LAN/WAN Tool 


Systems and 
Applications, 
and Distributed 
































Systems 
Compan Inc. 
Platforms Sun SPARC, PCs (Windows | PCs (Windows | Sun Micro- Digital, IBM, 
3.1, NT and stations, IBM, HP, Sun 
SGI, Windows | Windows 95), | Windows 95), | HP Micro-stations, 
NT UNIX, DEC, Macintosh PC’s 


(Windows NT 
and Windows 
95) 


Price 
Commercial $18,000.00 $48,500.00 $695.00 $23,500.00+ $40,800.00 
University $600.00 $3000.00 $350.00 $500.00 $500.00 


Accessibili 3 


Completeness 4 


Management 


Cost 2 


HP UX, SGI, 
AIX, Solaris 
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OPNET COMNET Ill EXTEND WORKBENCH G2 


MODELER 

Methods 

Reduction 

Documentation | 4 | CT UTC eC 3 
Ease Of Use 2 
Desigr 

Extendibili 5 
Of Results 

Fideli 5 
Flexibili 4 
Tateroperability j 
Miscellaneous 2 
Environment 

Support 

Reliabili 5 
Securi 3 
Software 4 
Tutorial 

Asdeanee | 

User Interface 4 
Validation 5 
Verification 5 
Non Weighted 106 87 100 97 95 
Totals 


Table 7-1. Comparison of Products. 


c: THE GRAND SOLUTION 

A quick total suggests that OPNET 1s the best overall software product. However, 
the criteria must be weighted when selecting a specific software product for the user’s 
needs. 

By weighting each evaluation criterion, a priority for the characteristics required 
for a mission is established. These weights may differ for each distinct task. An 
organization should give serious consideration to the weights based on their generic usage 


of the product (unless the organization can afford to purchase more than one product). 


This will provide the organization with a product that may not be the “best” for every 
instance, but will be the “best” for the organization’s overall usage. 

An example, a unit is looking for a software product that can model experimental 
architectures and systems in addition to communication and computer networks. The unit 
has no UNIX machines, mostly PCs, and a few Macintoshes. The operators are proficient 
in C Language, but troubleshooting assistance and user interface are considered key for 
the experimental design of the system. Frequency and terrain have no bearing on the 
systems to be modeled. Minimizing cost is very important to the unit. The unit may 
establish the following weights: Completeness - 1, Cost - 5, Experimental Design - 5, 
Extendibility - 4, Miscellaneous - 1, Security - 1, Software - 5, Troubleshooting 
Assistance - 5, User Interface - 5, and all other criteria may be a 3. Each weight is then 
multiplied by the product’s score to produce a weighted score. The weighted scores are 
totaled. The product with the highest weighted score’is the best modulation simulator 
based on the current priorities. In this example that would be Extend. Table 7-2 depicts 


the comparison example with weighted scores. 


OPNET | COMNETII | EXTEND | WORKBENCH G2 

MODELER 
General Use 5*4 = 20 
Platform 4*4 = 16 
Accessibili 3*3 = 9 
Completeness 1*4=4 


Configuration Bt) = 15 3*4=12 3*4 = 12 B75 = 15 375 =15 
Management 


Cost 5*4 = 20 5*5 = 25 5*3 = 15 5*2 = 10 


Methods 
Reduction 


Documentation 3*4 = 12 3*4 = 12 3*3 =9 
Ease Of Use 3*4 = 12 3*2 =6 


Experimental 5*4 =20 otzZ =10 Se —25 Ssh als) 3 — 5 
Design 


Extendibili A*4 = 16 A*4 = 16 A*5 = 20 A*5 = 20 


Extensibility 3*5 = 15 3*4 = 12 3*4 = 12 3*4 =12 
Of Results : 


Fideli 3*5 = 15 
Flexibili 5*4 = 20 
Interoperability 2*4 = 8 
Miscellaneous 1*2=2 


Operational 3*5 = 15 3*5 = 15 3*5 = 15 3*5 = 15 
Environment 
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OPNET COMNET Iii EXTEND WORKBENCH G2 
MODELER 


Support 

Reliabili 345 = 15 3*5'= 15 
Securi 1*3 =3 eee 
Software 5*4 = 20 
Tutorial 

Assistance 
User Interface 5*4 = 20 
Validation S56 15 
Verification Seal 
Non Weighted 369 306 378 348 344 
Totals 


Table 7-2. Example Weighted Comparison of Products. 


D. OVERALL COMPARISON 

Each of the five products analyzed in this thesis has its own special characteristics 
and capabilities. Why select one over the other? The functions and capabilities vary 
making some products more preferable for different applications and scenarios. 

CACI’s COMNET III is the best choice for modeling a WAN or LAN. COMNET 
If provides all the familiar computer network icons in a user friendly environment to 
enable the user to quickly model and analyze a computer network. COMNET II, 
however, does not model communication or information systems. Nor does COMNET Ii 
model the innards of a computer. 

MIL 3 developed OPNET Modeler which can model computers, communication 
and information systems from the network through the process and state levels. 
Numerous libraries are available to assist the modeler. OPNET’s primary disadvantage is 
its fixed run speed. This can cause extensive time delays during a simulation run of a 
detailed model. 

SES’ Workbench, like OPNET, can be used to model communication, computer, 
and information systems. In addition, Workbench can be used to model numerous other 
types of systems, to include queues, electric flow, water pollution, etc.. The ability to 
model more than just communication, computer and information systems provides an 


added flexibility to the modeler. 


Extend, developed by Imagine That Inc., is similar in its capabilities to 
Workbench, in that it can, also, model any type of system. Extend, however, provides a 
more user friendly environment for the modeler. Besides the documentation’s ease of 
use, the menu setup has a more logical flow. Workbench, through the use of sub- 
modules, can model to a smaller level of detail than Extend, and Workbench can handle 
larger sized models. 

G2, also similar to Workbench in its system modeling capabilities, is in a class by 
itself. G2 utilizes an object oriented modeling environment with a structured natural 
language. This product is a great selection for a beginner or non-programmer. Object 
oriented modeling enables the modeler to design the level of fidelity and complexity to be 
modeled. G2 includes a wide range of knowledge bases or libraries, though the labeling 
is puzzling. G2/object oriented modeling is not an easy product/modeling method to 
learn. 

It is essential to look at the capabilities and characteristics of the modeler(s). 
Programming skills can influence the product selection. All five of the products are icon 
and menu oriented. COMNET II provides the smallest programming interface, and G2 
has an easy to learn structured natural language. OPNET, Extend, and Workbench, each 
use a modified version of the C language. Workbench presents the user with the largest 
interface with the model’s code, and provides the user with the most programming 


opportunities. 
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VII. CONCLUSIONS 


A. SUMMARY 

Today’s technological advances are nearly overwhelming. Increasing bandwidths, 
increasing data rates, new protocols and more, all lead to new equipment available for the 
military. The increased usage of COTS products encourages units to upgrade and replace 
existing capabilities. 

The software products, analyzed in this thesis, can provide the user with a method 
for evaluating new equipment prior to its purchase. It, also, enables the user to optimize 
existing systems and networks, and to evaluate “what if” situations. 

Each product has individual characteristics which make it the best choice for a 
particular user in a specific situation. This thesis evaluated five modulation simulator 
software products based on twenty-five evaluation criteria. 

OPNET Modeler, by MIL 3 Inc., provides built-in familiar network icons. It 
provides the user with limited access to the code. The user can program entrance and exit 
criteria for low level states and processes. Access to code and debugging is in Proto-C. 
This can be troublesome for a non-programmer. 

COMNET II, from CACI Products Company, is easy to learn. It provides quick 
high to intermediate level network modeling with familiar built-in icons. 

Extend, developed by Imagine That Inc., is very user friendly. It does not contain 
network icons, but the user can build them. The product is flexible and can be used to 
model more than just networks, computers and communication equipment. 

Workbench, created by Scientific and Engineering Software Inc., permits users to 
access the model’s code. Network icons are not included, but can be created by the user 
or purchased for an additional cost. Access to the model’s code and debugging the model 
are all completed in a language similar to C. These can be difficult for a non- 
programmer. 

G2, by Gensym Corporation, provides a completely object oriented modeling 
environment. The documentation is very confusing. Programming is easy for non- 
programmers. It is completed in a structured natural language similar to the human 


language. 
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A user’s needs and modeling and programming capabilities should be evaluated 
prior to choosing a software product. A weighted criteria evaluation is recommended for 


selecting the correct software tool. 


B. APPLICATIONS WITHIN THE MILITARY 

The complex technology and the limited availability of troops and money 
provides an environment without the capability to fully test, evaluate, integrate, and 
analyze new equipment. Modulation simulators can be used to evaluate the integration of 
new equipment. New equipment can be modeled, tested and integrated for less money 
and with less troops than physically fielding a piece of equipment. Existing systems can 
be load-tested, reorganized, and reconnected. Outages and equipment losses can be 


simulated without the use of actual equipment and soldiers. 


C: RECOMMENDED FUTURE STUDIES 

The software tools evaluated in this thesis can be used to model virtually any 
computer, communication, or information system. Future studies could include modeling 
and evaluating existing or emerging command and control or communications systems. 

The growing inventory of modulation simulators has produced a concern for 
interface capabilities and limitations. This can be expanded to include uses and interfaces 
with virtual reality simulators. 

Additionally, the Naval Postgraduate School is upgrading its existing internal 
computer network. The current and future networks could be modeled and analyzed for 


shortcomings and improvements. 
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APPENDIX. DEFINITIONS OF ASSESSMENT MEASURES 


1. ACCESSIBILITY 
° Platforms supported 
e Conversion packages for switching to other platforms 
e Plans for support to any other platforms in the near future 
e PC version: DOS or UNIX based? 
2. COMPLETENESS 
e Interactively models computers, software and communications 
equipment 
e Includes or is capable of modeling current and new computer 
architectures 
e Includes or is capable of modeling current and new LANs / WANs 
e Includes or is capable of modeling current or new mass storage devices 
e Includes or is capable of modeling Intra- and Inter- machine 
communications 
e Includes or is capable of modeling military communication devices 
3. CONFIGURATION MANAGEMENT 
e Maturity of software 
e User view of the model 
4. COST 
e Military acquisition prices (What extras are included?) 
e Maintenance costs 
e Training costs 
e Upgrade costs 
5. DATA INPUT METHODS 
e Internal Methods 
e External Methods 
6. DATA REDUCTION 
e How quickly 
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e What methods (Poisson, Wiebull, etc.) 
7. DOCUMENTATION 
e Requirements specification 
e Design document 
e Conceptual model document 
e Tutorial 
e User’s Manual 
8. EASE OF USE 
e Support for computer/network modeling 
e Support for software modeling (pre-analysis sizing, communications, 
functional flow, data flow, CPU loading, machine instruction 
specification, etc.) 
e Support for data channeling modeling 
e Support for communications traffic modeling 
e Ease of learning 
e Manuals readily available and easy to understand 
e Set up time 
e Runtime 
* Can it be varied 
* Typical run duration 
9. EXPERIMENTAL DESIGN 
e Supports Monte Carlo 
e Supports algorithms 
e Iterative runs 
10. EXTENDIBILITY 
e Can tool be readily modified? 
e Design or programmatic? 
e What language? 
11. EXTENSIBILITY OF RESULTS 
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e Alternative computer architectures 
* Extension to hypothetical architectures 
* Extension to architectures that may be performance tested 
* Extension to architecture that are not available for testing 
e Alternative data channels 
* Extension to hypothetical architectures 
* Extension to architectures that may be performance tested 
* Extension to architecture that are not available for testing 
e Alternative software descriptions 
e Alternative communication requirements 
12. FIDELITY 
e Tool provides sufficient fidelity to ensure output accuracy across the 
tool architecture 
13. FLEXIBILITY 
e Can nodes move on line / off line? (How implemented) 
e How quickly reconfigure? 
14. INTEROPERABILITY 
e Known interface capabilities (software / hardware) 
e Interfaces under development 
e Partial interfaces available (just the scenario?) 
15. MISCELLANEOUS 
e Terrain 
* ITs it available? 
* Can equipment be tied to terrain? 
e Frequency Management capabilities 
e Outages 
* Planned / unplanned outages (How implemented?) 
e Additional features 
16. OPERATIONAL ENVIRONMENT 


aS 


e System 
e Portable 
e Availability 
17. POST ANALYSIS SUPPORT 
e Reports (Variety) 
e How quickly 
e Graphs (Variety) 
18. RELIABILITY 
e Will it work every time producing the same results? 
19. SECURITY 
e Type of security (Classified or User) 
e Labeling 
e Passwords 
20. SOFTWARE 
e Process maturity 
e Architecture 
e Complexity 
e Quality 
e Reliability 
e Sufficiency of testing 
e Modularity or Object Oriented Design 
e Maintainability 
21. TRAINING / TUTORIAL 
e Is it understandable? User friendly? 
e Is it complete? 
e How proficient is user after training? (overview, beginner, master) 
22. TROUBLESHOOTING ASSISTANCE 
e Is animation available 


e Ability to view code (Can you see inside the model?) 
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e 1-800 help lines 

e Frequent problems list / lessons learned 
23. USER INTERFACE 

e Type (Windows, C programming, UNIX programming, etc.) 

e Quality 

e Effectiveness 
24. VALIDATION 

e Degree of accuracy 

e Variable (quick answer, intermediate answer, detailed answer) 
25. VERIFICATION 


e Of tool design and functions (How do we know it works?) 
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